Counterfactual fairness (Kushner et al., 2017)

A predictor \(\hat{Y}\) is counterfactually fair if, for any individual with observed attributes \(A = a\) (protected attribute) and \(X = x\) (remaining attributes), and for any other possible value \(a'\) of \(A\).

\[\begin{align} \begin{split} P\left( \hat{Y}_{A \leftarrow a} = y \mid A = a, X = x \right) &= P\left( \hat{Y}_{A \leftarrow a'} = y \mid A = a, X = x \right), \\ &\quad \text{for all } y. \label{eq:counterfactual} \end{split} \end{align}\]

Where:

• \(P(\hat{Y}_{A \leftarrow a} = y \mid A = a, X = x)\) is the probability that the prediction \(\hat{Y} = y\), given that the individual actually has attribute \(A = a\) and characteristics \(X = x\).

• \(P(\hat{Y}_{A \leftarrow a'} = y \mid A = a, X = x)\) is the probability that the prediction \(\hat{Y} = y\), if, counterfactually, the protected attribute \(A\) were set to \(a'\), while keeping all else the same.

Metrics

1. Sentiment analysis
   • SiEBERT - a general purpose, pre-trained, binary sentiment analysis model.
   • Regard - a pre-trained metric was designed for the purpose of evaluating gender bias across texts.

\[\begin{align*} \text{sentiment}_{ij} &= \beta_0 + \beta_1 \cdot \text{model}_i + \beta_2 \cdot \text{gender}_j \\ &\quad + \beta_3 \cdot (\text{model}_i \times \text{gender}_j) + \beta_4 \cdot \text{max_tokens}_i \\ &\quad + u_{0j} + u_{1j} \cdot \text{model}_i + \epsilon_{ij} \label{eq:summarieslmm} \end{align*}\]

2. Counts of words, themes
   • \(\chi^2\) test
   • Poisson regression

\[\begin{align*} \text{count}_{i} &= \beta_0 + \beta_1 \cdot \text{gender}_i + \beta_2 \cdot \text{max_tokens}_i \\ &\quad + \beta_3 \cdot \text{doc_id}_i + \epsilon_i \label{eq:worddtregression} \end{align*}\]

Sentiment analysis: estimated marginal means (female - male)

Frequency of themes

Word counts: Gemma